{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <b>Object Detection with AutoML Vision</b>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <b>Learning Objectives</b> ##\n",
    "\n",
    "1. Learn how to create and import an image dataset to AutoML Vision\n",
    "1. Learn how to train an AutoML object detection model\n",
    "1. Learn how to evaluate a model trained with AutoML\n",
    "1. Learn how to deploy a model trained with AutoML\n",
    "1. Learn how to predict on new test data with AutoML\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we will use AutoML Vision Object Detection to train a machine learning model capable of detecting multiple objects in a given image and provides information about the objects and their location within the image.\n",
    "\n",
    "We will start by creating a dataset for AutoML Vision and then import a publicly available set of images into it. After that we will train, evaluate and deploy the AutoML model trained for this dataset. Ultimately we show how to send prediction requests to our model through the deployed API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <b>AutoML Vision Setup</b> ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we begin make sure you have [created a project on the GCP Console](https://cloud.google.com/vision/automl/object-detection/docs/before-you-begin) and enabled the AutoML and Cloud Storage APIs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <b> Install AutoML and Cloud Storage package </b> ###\n",
    "<b>Caution: Run the following command and restart the kernel afterwards.</b>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pip freeze | grep google-cloud-automl==1.0.1 || pip install google-cloud-automl==1.0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pip freeze | grep google-cloud-storage==1.27.0 || pip install google-cloud-storage==1.27.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "from google.cloud import automl\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "### <b>Set the correct environment variables </b> ###\n",
    "The following variables should be updated according to your own environment:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PROJECT_ID = \"YOUR_PROJECT_ID\" # Replace with your PROJECT ID\n",
    "SERVICE_ACCOUNT = \"YOUR_SERVICE_ACCOUNT_NAME\" # Replace with a name of your choice\n",
    "ZONE = \"us-central1\"# Make sure the zone is set to \"us-central1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "The following variables are computed from the one you set above, and should not be modified:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PWD = os.path.abspath(os.path.curdir)\n",
    "\n",
    "SERVICE_KEY_PATH = os.path.join(PWD, \"{0}.json\".format(SERVICE_ACCOUNT))\n",
    "SERVICE_ACCOUNT_EMAIL=\"{0}@{1}.iam.gserviceaccount.com\".format(SERVICE_ACCOUNT, PROJECT_ID)\n",
    "print(SERVICE_ACCOUNT_EMAIL)\n",
    "print(PROJECT_ID)\n",
    "\n",
    "# Exporting the variables into the environment to make them available to all the subsequent cells\n",
    "os.environ[\"PROJECT_ID\"] = PROJECT_ID\n",
    "os.environ[\"SERVICE_ACCOUNT\"] = SERVICE_ACCOUNT\n",
    "os.environ[\"SERVICE_KEY_PATH\"] = SERVICE_KEY_PATH\n",
    "os.environ[\"SERVICE_ACCOUNT_EMAIL\"] = SERVICE_ACCOUNT_EMAIL\n",
    "os.environ[\"ZONE\"] = ZONE\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "### <b>Switching the right project and zone</b> ###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "gcloud config set project $PROJECT_ID\n",
    "gcloud config set compute/region $ZONE\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "### <b>Create a service account and generate service key</b> ###\n",
    "\n",
    "\n",
    "Before we can run our program we need to get it authenticated. For that, we first need to generate a service account.\n",
    "A service account is a special type of Google account intended for non-human users (i.e., services) that need to authenticate and be authorized to access data through Google APIs (in our case the AutoML and Cloud Storage API). After the service account has been created it needs to be associated with a service account key, which is a json file holding everything that the client needs to authenticate with the service endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "gcloud iam service-accounts list | grep $SERVICE_ACCOUNT ||\n",
    "gcloud iam service-accounts create $SERVICE_ACCOUNT\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "test -f $SERVICE_KEY_PATH || \n",
    "gcloud iam service-accounts keys create $SERVICE_KEY_PATH \\\n",
    "  --iam-account $SERVICE_ACCOUNT_EMAIL\n",
    "\n",
    "echo \"Service key: $(ls $SERVICE_KEY_PATH)\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "### <b>Make the key available to google clients for authentication</b> ###\n",
    "AutoML API will check this environement variable to see where the key is located and use it to authenticate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = SERVICE_KEY_PATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "### <b>Grant service account required role permissions</b> ###\n",
    "\n",
    "After we have created our service account and associated it with the service key we need to assign some permissions through a role. For this example we only need to grant our service account the automl and storage admin role so it has permission to complete specific actions on the resources of your project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "gcloud projects add-iam-policy-binding $PROJECT_ID \\\n",
    " --member \"serviceAccount:$SERVICE_ACCOUNT_EMAIL\" \\\n",
    " --role \"roles/automl.admin\" \\\n",
    " --role \"roles/storage.admin\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "## <b>Step 1: Preparing and formatting training data</b> ##\n",
    "\n",
    "The first step in creating a custom model with the AutoML Vision is to prepare the training data. In this case the training dataset that is composed of images along with information identifying the location (through bounding boxes coordinates) and type of objects (through labels) in the images. \n",
    "Here are some constraints some general rules for preparing an AutoML object detection dataset:\n",
    "\n",
    "* The following image formats are supported: JPEG, PNG, GIF, BMP, or ICO. Maximum file size is 30MB per image.\n",
    "\n",
    "* AutoML Vision models can not generally predict labels that humans can't assign. So, if a human can't be trained to assign labels by looking at the image for 1-2 seconds, the model likely can't be trained to do it either.\n",
    "\n",
    "* It is recommended to have about 1000 training images per label (i.e. object type you want to detect in the images). For each label you must have at least 10 images, each with at least one annotation (bounding box and the label). In general, the more images per label you have the better your model will perform."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "### <b>Training vs. evaluation datasets</b> ###\n",
    "\n",
    "When training machine learning models you typically divide the dataset usually into three separate datasets:\n",
    "\n",
    "1. a training dataset\n",
    "1. a validation dataset\n",
    "1. a test dataset\n",
    "\n",
    "A training dataset is used to build a model. The model being trained tries multiple parameters while searching for patterns in the training data. During the process of pattern identification, AutoML Vision Object Detection uses the validation dataset to test the parameters of the model. AutoML Vision Object Detection chooses the best-performing algorithms and patterns from all options identified during the training stage.\n",
    "\n",
    "After the best performing algorithms and patterns have been identified, they are tested for error rate, quality, and accuracy using the test dataset.\n",
    "\n",
    "Both a validation and a test dataset are used in order to avoid bias in the model. During the validation stage, optimal model parameters are used. Using these optimal model parameters can result in biased metrics. Using the test dataset to assess the quality of the model after the validation stage provides the training process with an unbiased assessment of the quality of the model.\n",
    "\n",
    "\n",
    "By default, AutoML Vision Object Detection splits your dataset randomly into 3 separate sets (you don't need to do it yourself!):\n",
    "\n",
    "* 80% of images are used for training.\n",
    "* 10% of images are used for hyper-parameter tuning and/or to decide when to stop training.\n",
    "* 10% of images are used for evaluating the model. These images are not used in training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "### <b>Create a CSV file with image URIs and labels</b> ###\n",
    "\n",
    "Once your image files have been uploaded to a Cloud Storage bucket (`gs://bucket-name-vcm`), you must create a CSV file that lists all of the URI of the uploaded images, along with bounding box information and the object labels. The dataset will contain one row per bounding box in the image, so an image that has two bounding boxes will have two corresponding rows in the CSV file sharing the same image URI. The CSV file can have any filename, must be in the same bucket as your image files, must be UTF-8 encoded, and must end with a `.csv` extension. \n",
    "\n",
    "\n",
    "In the example below, rows 1 and 2 reference the same image that has 2 annotations \n",
    "`(car,0.1,0.1,,,0.3,0.3,,)` and  `(bike,.7,.6,,,.8,.9,,)`. The first element of the annotation\n",
    "is the object label in the bounding box, while the rest are the coordinates of the bounding box\n",
    "within the image (see below for details).\n",
    "\n",
    "\n",
    "Row 3 refers to an image that has only 1 annotation `(car,0.1,0.1,0.2,0.1,0.2,0.3,0.1,0.3)`, while row 4 references an image with no annotations.\n",
    "\n",
    "The first column corresponds to the data split, the second column to the image URI, and the last columns hold the annotations.\n",
    "\n",
    "**Example:**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "TRAIN,gs://folder/image1.png,car,0.1,0.1,,,0.3,0.3,,\n",
    "TRAIN,gs://folder/image1.png,bike,.7,.6,,,.8,.9,,\n",
    "UNASSIGNED,gs://folder/im2.png,car,0.1,0.1,0.2,0.1,0.2,0.3,0.1,0.3\n",
    "TEST,gs://folder/im3.png,,,,,,,,,\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each row above has these columns:\n",
    "`\n",
    "1. <b>Which dataset is the content in the row being assigned to.</b> - `TRAIN`, `VALIDATE`, `TEST` or `UNASSIGNED`\n",
    "1. <b>What content is being annotated.</b> - It contains the GCS URI for the image\n",
    "1. <b>A label that identifies how the object is categorized.\n",
    "1. <b>A bounding box for an object in the image.</b>\n",
    "    \n",
    "\n",
    "The **bounding box** for an object can be specified in two ways:\n",
    "    \n",
    "    *  with only 2 vertices (consisting of a set of x and y coordinates) if they are diagonally opposite points of the rectangle \n",
    "```  \n",
    "(x_relative_min,y_relative_min,,,x_relative_max,y_relative_max,,)\n",
    "```   \n",
    "    * with all 4 vertices\n",
    "```    \n",
    "(x_relative_min,y_relative_min,x_relative_max,y_relative_min,x_relative_max,y_relative_max,x_relative_min,y_relative_max)\n",
    "```\n",
    "    \n",
    "Each vertex is specified by x, y coordinate values. These coordinates must be a float in the 0 to 1 range, where 0 represents the minimum x or y value, and 1 represents the greatest x or y value.\n",
    "\n",
    "For example, `(0,0)` represents the top left corner, and `(1,1)` represents the bottom right corner; a bounding box for the entire image is expressed as `(0,0,,,1,1,,)`, or `(0,0,1,0,1,1,0,1)`.\n",
    "\n",
    "AutoML API does not require a specific vertex ordering. Additionally, if 4 specified vertices don't form a rectangle parallel to image edges, AutoML API calculates and uses vertices that do form such a rectangle."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generating a CSV file for unlabeled images stored in Cloud Storage ###\n",
    "\n",
    "If you already have unlabeled images uploaded to Cloud Storage and would like to generate a CSV pointing to them, run this code in Cloud Shell:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "for f in $(gcloud storage ls gs://YOUR_BUCKET/YOUR_IMAGES_FOLDER/);\n",    "do echo UNASSIGNED,$f;\n",
    "done >> labels.csv;\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then copy the generated CSV file into a Google Storage Bucket:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```gcloud storage cp labels.csv gs://YOUR_BUCKET/labels.csv```"   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then after uploading the images to AutoML Object Detection, you can use Cloud Vision API's [Object Localizer](https://cloud.google.com/vision/docs/object-localizer) feature to help build your dataset by getting more generalized labels and bounding boxes for objects in an image."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "## <b>Step 2: Create a dataset</b> ##\n",
    "\n",
    "Next step is to create and name an empty dataset that will eventually hold the training data for the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATASET_NAME = \"salad_dataset\" # Replace with desired dataset name\n",
    "\n",
    "client = automl.AutoMlClient()\n",
    "\n",
    "# A resource that represents Google Cloud Platform location.\n",
    "project_location = client.location_path(PROJECT_ID, ZONE)\n",
    "metadata = automl.types.ImageObjectDetectionDatasetMetadata()\n",
    "dataset = automl.types.Dataset(\n",
    "    display_name=display_name,\n",
    "    image_object_detection_dataset_metadata=metadata,\n",
    ")\n",
    "\n",
    "# Create a dataset with the dataset metadata in the region.\n",
    "response = client.create_dataset(project_location, dataset)\n",
    "\n",
    "created_dataset = response.result()\n",
    "\n",
    "# Display the dataset information\n",
    "print(\"Dataset name: {}\".format(created_dataset.name))\n",
    "print(\"Dataset id: {}\".format(created_dataset.name.split(\"/\")[-1]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "## <b>Step 3: Import images into a dataset</b> ##\n",
    "\n",
    "\n",
    "After you have created a dataset, prepared and formated your training data, it's time to import that training data into our created dataset.\n",
    "\n",
    "In this notebook we will use a publicly available \"Salads\" training dataset that is located at `gs://cloud-ml-data/img/openimage/csv/salads_ml_use.csv`.\n",
    "\n",
    "This dataset contains images of salads with bounding boxes and labels around their ingredients (e.g., tomato, seafood, etc.).\n",
    "So the model we will train will be able to take as input the image of a salad and determine the ingredients composing the salad\n",
    "as well as the location of the ingredients on the salad image.\n",
    "\n",
    "Please note the import might take a couple of minutes to finish depending on the file size.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATASET_ID = format(created_dataset.name.split(\"/\")[-1])\n",
    "DATASET_URI = \"gs://cloud-ml-data/img/openimage/csv/salads_ml_use.csv\" \n",
    "\n",
    "# Get the full path of the dataset.\n",
    "dataset_full_id = client.dataset_path(\n",
    "    PROJECT_ID, ZONE, DATASET_ID\n",
    ")\n",
    "# Get the multiple Google Cloud Storage URIs\n",
    "input_uris = path.split(\",\")\n",
    "gcs_source = automl.types.GcsSource(input_uris=input_uris)\n",
    "input_config = automl.types.InputConfig(gcs_source=gcs_source)\n",
    "\n",
    "# Import data from the input URI\n",
    "response = client.import_data(dataset_full_id, input_config)\n",
    "\n",
    "print(\"Processing import...\")\n",
    "print(\"Data imported. {}\".format(response.result()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "## <b>Step 4: Train your AutoML Vision model</b> ##\n",
    "\n",
    "Once you are happy with your created dataset you can proceed with training the model. <i>Please note</i> - training time takes approximately <b>1-3h</b>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_NAME = \"salads\" # Replace with desired model name\n",
    "\n",
    "# A resource that represents Google Cloud Platform location.\n",
    "\n",
    "project_location = client.location_path(PROJECT_ID, ZONE)\n",
    "\n",
    "# Leave model unset to use the default base model provided by Google\n",
    "# train_budget_milli_node_hours: The actual train_cost will be equal or\n",
    "# less than this value.\n",
    "# https://cloud.google.com/automl/docs/reference/rpc/google.cloud.automl.v1#imageobjectdetectionmodelmetadata\n",
    "training_metadata = automl.types.ImageObjectDetectionModelMetadata(\n",
    "    train_budget_milli_node_hours=24000\n",
    ")\n",
    "model = automl.types.Model(\n",
    "    display_name=display_name,\n",
    "    dataset_id=dataset_id,\n",
    "    image_object_detection_model_metadata=metadata,\n",
    ")\n",
    "\n",
    "# Create a model with the model metadata in the region.\n",
    "training_results = client.create_model(project_location, model)\n",
    "\n",
    "print(\"Training operation name: {}\".format(response.operation.name))\n",
    "print(\"Training started...\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "### <b>Information about the trained model</b> ###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_ID = format(model.name.split(\"/\")[-1])\n",
    "\n",
    "# Get the full path of the model.\n",
    "model_full_id = client.model_path(PROJECT_ID, ZONE, MODEL_ID)\n",
    "model = client.get_model(model_full_id)\n",
    "\n",
    "# Retrieve deployment state.\n",
    "if model.deployment_state == automl.enums.Model.DeploymentState.DEPLOYED:\n",
    "    deployment_state = \"deployed\"\n",
    "else:\n",
    "    deployment_state = \"undeployed\"\n",
    "\n",
    "# Display the model information.\n",
    "print(\"Model name: {}\".format(model.name))\n",
    "print(\"Model id: {}\".format(model.name.split(\"/\")[-1]))\n",
    "print(\"Model display name: {}\".format(model.display_name))\n",
    "print(\"Model create time:\")\n",
    "print(\"\\tseconds: {}\".format(model.create_time.seconds))\n",
    "print(\"\\tnanos: {}\".format(model.create_time.nanos))\n",
    "print(\"Model deployment state: {}\".format(deployment_state))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "## <b>Step 5: Evaluate the model</b> ##\n",
    "\n",
    "After training a model, Cloud AutoML Vision Object Detection uses images from the TEST image set to evaluate the quality and accuracy of the new model.\n",
    "\n",
    "It provides an aggregate set of evaluation metrics indicating how well the model performs overall, as well as evaluation metrics for each category label, indicating how well the model performs for that label.\n",
    "\n",
    "By running the cell below you can list evaluation metrics for that model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"List of model evaluations:\")\n",
    "for evaluation in client.list_model_evaluations(model_full_id, \"\"):\n",
    "    print(\"Model evaluation name: {}\".format(evaluation.name))\n",
    "    print(\n",
    "        \"Model annotation spec id: {}\".format(\n",
    "            evaluation.annotation_spec_id\n",
    "        )\n",
    "    )\n",
    "    print(\"Create Time:\")\n",
    "    print(\"\\tseconds: {}\".format(evaluation.create_time.seconds))\n",
    "    print(\"\\tnanos: {}\".format(evaluation.create_time.nanos / 1e9))\n",
    "    print(\n",
    "        \"Evaluation example count: {}\".format(\n",
    "            evaluation.evaluated_example_count\n",
    "        )\n",
    "    )\n",
    "    print(\n",
    "        \"Object detection model evaluation metrics: {}\\n\\n\".format(\n",
    "            evaluation.image_object_detection_evaluation_metrics\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "## <b>Step 6: Deploy the model</b> ##\n",
    "\n",
    "Once we are happy with the performance of our trained model, we can deploy it so that it will be\n",
    "available for predictions through an API. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = client.deploy_model(model_full_id)\n",
    "\n",
    "print(\"Model deployment finished. {}\".format(response.result()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "## <b>Step 7: Send prediction request</b> ##\n",
    "\n",
    "In this example we will invoke an individual prediction from an image that is stored in our project's Cloud storage bucket.\n",
    "Object detection models output many bounding boxes for an input image. For the output we are expecting that each box comes with:\n",
    "1. a label and \n",
    "1. a score of confidence.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TEST_IMAGE_PATH = \"gs://your-bucket-name-vcm/your-folder-name/your-image.jpg\" # Replace with a Cloud storage bucket uploaded image of your choice\n",
    "\n",
    "prediction_client = automl.PredictionServiceClient()\n",
    "\n",
    "# Read the file.\n",
    "with tf.io.gfile.GFile(TEST_IMAGE_PATH, \"rb\") as content_file:\n",
    "    content = content_file.read()\n",
    "\n",
    "image = automl.types.Image(image_bytes=content)\n",
    "payload = automl.types.ExamplePayload(image=image)\n",
    "\n",
    "# params is additional domain-specific parameters.\n",
    "# score_threshold is used to filter the result\n",
    "# https://cloud.google.com/automl/docs/reference/rpc/google.cloud.automl.v1#predictrequest\n",
    "params = {\"score_threshold\": \"0.8\"}\n",
    "\n",
    "response = prediction_client.predict(model_full_id, payload, params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the response object from the deployed model, we can inspect its predictions (i.e., the\n",
    "bounding boxes and objects that the model has detected from the images we sent to it in the cell above):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Prediction results:\")\n",
    "for result in response.payload:\n",
    "    print(\"Predicted class name: {}\".format(result.display_name))\n",
    "    print(\n",
    "        \"Predicted class score: {}\".format(\n",
    "            result.image_object_detection.score\n",
    "        )\n",
    "    )\n",
    "    bounding_box = result.image_object_detection.bounding_box\n",
    "    print(\"Normalized Vertices:\")\n",
    "    for vertex in bounding_box.normalized_vertices:\n",
    "        print(\"\\tX: {}, Y: {}\".format(vertex.x, vertex.y))"
   ]
  }
 ],
 "metadata": {
  "environment": {
   "name": "tf2-gpu.2-1.m47",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-1:m47"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
